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SUMMARY 


Ij»>  ttrii  paper  w  afpty  Ihe  functional  aquation 
technique  of  the  theory  of  dynanlc  programming 
to  the  theory  of  equipment  replacement. 
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NOTES  IN  THE  THEORY  OF  DYNAMIC  PROORAMMINO  -  III 
EQUIPMENT  REPLACEMENT  POLICY 

by 

Richard  Bellman 


H .  Introduction 

A  problem  of  Interest  In  Industrial  economics  Is  that  of 
determining  the  optimal  procedure  for  the  replacement  of  old 
equipment  by  new.  This  problem  has  been  treated  by  Alchlan  In 
a  very  Interesting  monograph,  [l] .  In  this  paper  we  wish  to 
show  hoM  the  problem  may  be  approached  by  the  functional  equa— 
tlon  technique  of  the  theory  of  dynamic  programming.  We 
shall  first  consider  the  simple  case  in  which  there  is  no 
technological  Improvement,  and  show  that  this  problem  may  be 
resolved  very  easily.  Then  we  shall  consider  the  more  diffi¬ 
cult  and  Important  case  where  there  Is  technological  Improve¬ 
ment,  and  derive  a  functional  equation  for  the  solution  of 
this  problem.  A  more  detailed  discussion  of  this  equation 
will  be  given  In  a  further  cormnunlcatlon. 

%2 .  The  Process 

Let  us  assume  that  we  have  a  piece  of  equipment  with  the 
following  operating  characteristics: 


An  exposition  of  the  theory  and  an  extensive  bibliography 
may  be  found  In  [2] . 


Output 


Upkeep 
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t  -  Age  of  Machine 


Here  t  le  the  age  of  the  machine  measured  in  appropriate  units. 
Furthermore,  let  us  asaume  that  the  trade-in  value  of  the  machine 
as  a  function  of  time  ^las  the  form 


where  p  is  the  purchase  price  of  a  new  machine. 

Given  this  Information,  the  problem  ia  to  determine  the 
age  at  which  the  machine  should  be  replaced.  We  shall  assume 
that  we  are  considering  an  unbounded  process  where  we  keep 
using  machines  and  replacing  machines  and  so  forth.  Further¬ 
more,  let  us  assume  that  machines  can  only  be  replaced  at 
times  t  -  1,  2,  ...,  and  that  there  is  immediate  delivery. 


In  order  to  keep  the  over-«ll  return  finite,  let  uo  dis¬ 
count  the  return  one  stage  ahead  by  a  factor  a,  where  0  <  a  <  1. 

Under  these  conditions,  we  wish  to  determine  the  replace¬ 
ment  policy  wnlch  maximizes  the  ovei>-all  discounted  return. 

53*  Mathematical  Formulation 

It  1b  clear  that  the  only  state  variable  1»  t,  the  age  of 
the  machine.  Let  ub  then  define 

(1)  f(t)  -  overall  return  from  a  machine  of  age  t  employing 

an  optimal  replacement  policy. 

At  each  time  t,  we  have  one  of  two  courses  of  action  open.  We 
may  either  keep  the  machine  for  another  time  period,  or  we  may 
purchase  a  new  machine.  In  the  first  case,  we  see  that  f(t) 
satisfies  the  equation 

(2)  » 

and  In  the  second  case 

(3)  fp(t)  -  s(t}  -  p  r(0)  -  u(0)  ^  af(l)  . 

Hence  the  functional  equation  for  f(t)  Is 

K:  r(t)  -  u(t)  ♦  af(t-*-l)  *] 

(4 )  f  (t )  -  Max 

^P;  8(t)  -  p  r(0)  -  u(0)  af(l)  J 

♦  4.  Solution 

An  optimal  policy  will  have  the  form:  keep  a  new  machine 
until  It  Is  T  years  old  and  then  purchase  a  new  one.  This  yields 

the  following  system  of  equations 
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(1)  f(0)  -  n(0)  4  af(l) 

f(l)  -  n(l)  -f  af(2)  , 

f(T-l)  -  n(T-l)  ♦  af(T)  , 

f(T)  -  ;j(T)  -  p  >  n(0)  +  af(l)  , 


where  we  have  set  n(t)  •  r(t)  -  u(t).  It  li  clear  from  (3.  4) 
that  T  /  0,  Blnce  p  >  8(0).  Solving  for  f(l)  recurrently,  we 
obtain 

(2)  f(l)  -  n(l)-n(n(2)-»-af  (3))  -  n(l  )-^an(2)*i-a»f  (3) 

-  n(l)-*-an(2)>.  .  .4-a^~^n(T-l  )-fa^”^f  (T) 

-  [  n(l  )-*-an(2)*»- . .  .-fa'^“^n(T-l )  ] 

-  a(T)-p>n(0)4-af(l)  ] 


Hence 


(3) 


[  n(l  )*».an(2)-»-. .  .+a'^“^n(T-l )‘^n(0)a^'“^  ]-».a*’’‘‘  [  •(T)-p  ] 
f(l) - ^ - - 


T-l 


T-l 


1  -  a 


Since  T  Is  to  be  choeen  to  maxlalze  r(0),  and  hence  f(l},  we  see 
that  the  optimal  value  of  T  le  furnished  by  the  value  of  T  which 
yields  the  absolute  maximum  of  the  right  side  of  (3)  above  for 
T  -  1,  2,  ...  .  Let  us  call  this  maximum  T. 


This  Is  a  very  simple  computatloit  to  perform  given  the 
preceding  curves. 

15.  Over— Age  Machines 


Actually  we  do  not  have  the  complete  solution  to  the 
problem,  since  we  do  not  know  what  to  do  with  an  over— age 
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machlne,  one  whoee  age  1b  greater  than  T.  It  Is  not  clear  that 
In  this  case  the  optimal  policy  Involves  purchaelng  a  nen*  machine 
Immediately,  and,  as  a  matter  of  fact.  It  Is  not  unrestrictedly 
true.  To  determine  the  optimal  procedure,  we  use  the  relation 
In  (3-^)  for  t  >  T  1 .  'There  will  then  be  a  eecond  critical 
ag*  T»,  determined  very  much  the  eame  way  ae  above,  at  which  we 
purchase  a  new  machine  starting  wl„h  this  over— age  machine.  It 
1b  clear  that  by  suitably  altering  n(t)  and  s(t)  for  large  values 
of  t,  we  can  have  as  many  switchover  points  as  we  wish. 

§6  .  Technological  Imj^rovement 

Let  us  now  consider  the  more  realistic  situation  where 
Improved  operating  techniques  will  Increase  the  future  return 
from  the  same  machine,  and  new  machines  will  be  developed.  There 
are  now  two  state  variables,  t,  the  age  of  the  machine,  and  r, 
absolute  time. 

Let  us  define  as  above 

(1)  l’(r,t)  -  overall  return  obtained  from  a  machine  of 

age  t  at  time  t,  using  an  optimal  replace¬ 
ment  policy. 

The  functional  equation  is 

I”  K;  r(  T  ,t)-u(  T"  ,t  )-faf  (  f  4-1  ,t-fl ) 

b(  t  ,t)-p4-r(  T,0)-u(  t,0) 

4.af(r4.1,l) 


(2) 


1  (  t  ,  t )  -  Max 


P: 
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wher«  the  graphs  given  In  |l  typify  families  of  such  graphs 
dependent  upon  the  absolute  time.  This  equation  is  more  diffi¬ 
cult  to  treat  than  the  one  above.  Here  the  method  of  successive 
approximations  based  upon  an  initial  policy  space  approximation 
Mill  Mcrk  mrell.  We  shall  discuss  this  in  a  further  communication. 
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